Author Topic: Get Pixel Color  (Read 1936 times)

Exergist

  • Moderator
  • Sr. Member
  • *****
  • Posts: 325
  • Can you dig it?
Get Pixel Color
« on: August 16, 2017, 11:08:13 am »
I've seen a few questions/requests on the forum about getting pixel information with VoiceAttack. Gary has explained that this is not possible with VA's native functionality. However with the magic of C# and VA's support for inline functions we can come up with a command that captures the RGB (and hex) color values for the pixel under the mouse cursor. However, I'm going to echo Gary's comment that using this method may be considered a TOS violation depending on the application in use, so please keep that in mind.

VA Command Code:
Code: [Select]
// Obtain mouse cursor's X coordinate
Set integer [mouseX] value to the converted value of {MOUSESCREENX}
// Obtain mouse cursor's Y coordinate
Set integer [mouseY] value to the converted value of {MOUSESCREENY}
// Output cursor coordinates
Write '[Black] Cursor Coordinates = ({INT:mouseX} , {INT:mouseY})' to log
// C# function to obtain RGB and hexadecimal color values from screen pixel at cursor's X and Y coordinates
Inline C# Function: Get Pixel Color, wait until execution finishes
// Output individual RGB values
Write '[Red] Red Value = {INT:RedValue}' to log
Write '[Green] Green Value = {INT:GreenValue}' to log
Write '[Blue] Blue Value = {INT:BlueValue}' to log
Write '[Gray] Hex Value = {TXT:HexValue}' to log

Inline Function Code:
Referenced Assemblies: System.dll; System.Drawing.dll
Code: [Select]
using System;
using System.Drawing;
using System.Runtime.InteropServices;

public class VAInline
{
public void main()
{       
// Retrieve (X,Y) coordinates from VA variables
int Xpos = (int)VA.GetInt("mouseX");
int Ypos = (int)VA.GetInt("mouseY");

// Get pixel color data from (X,Y) coordinates
Color myColor = GetPixelColor(Xpos, Ypos);

// Store RGB color values
int r = myColor.R;
int g = myColor.G;
int b = myColor.B;

// Store HEX color value
string hex = myColor.R.ToString("X2") + myColor.G.ToString("X2") + myColor.B.ToString("X2");

// Send RGB data back to VA variables
VA.SetInt("RedValue", r);
VA.SetInt("GreenValue", g);
VA.SetInt("BlueValue", b);
VA.SetText("HexValue", hex);
}

// Function for retrieving pixel color at (x,y) screen coordinate
static public System.Drawing.Color GetPixelColor(int x, int y)
{
IntPtr hdc = GetDC(IntPtr.Zero);
uint pixel = GetPixel(hdc, x, y);
ReleaseDC(IntPtr.Zero, hdc);
Color color = Color.FromArgb((int)(pixel & 0x000000FF),
(int)(pixel & 0x0000FF00) >> 8,
(int)(pixel & 0x00FF0000) >> 16);
return color;
}

// Imported external functions
[DllImport("user32.dll")]
static extern IntPtr GetDC(IntPtr hwnd);
[DllImport("user32.dll")]
static extern Int32 ReleaseDC(IntPtr hwnd, IntPtr hdc);
[DllImport("gdi32.dll")]
static extern uint GetPixel(IntPtr hdc, int nXPos, int nYPos);
}

// References:
// https://stackoverflow.com/questions/753132/how-do-i-get-the-colour-of-a-pixel-at-x-y-using-c
// https://stackoverflow.com/questions/13354892/converting-from-rgb-ints-to-hex

To validate the command, let's turn to this RGB example at ColorPicker.com. Moving the cursor inside the small blue box (like as the red arrow indicates below) representing the color RGB = (85, 176, 240) and activating the command does indeed show that VA can now (indirectly) detect and capture the pixel color. Note that the screen coordinates in the below example show positive and negative values since this was captured on a multi-monitor setup.



I've included all of the above command and function code in the attached profile. For this example I've chosen to simply output the RGB and hex data, but I'm sure there are all kinds of interesting things folks can do with this data in VA. For instance this doesn't have to be applied to a single pixel - you could use a loop to check an array of pixels surrounding an onscreen point (or cursor position). Feel free to designate the screen (X,Y) coordinates however you see fit. Enjoy! :)

Edit: added code and output for HEX color value corresponding to the RGB
Edit: minor formatting updates
« Last Edit: June 13, 2018, 07:01:48 am by Exergist »

Tumbler

  • Newbie
  • *
  • Posts: 10
Re: Get Pixel Color
« Reply #1 on: June 20, 2018, 11:01:48 am »
Exergist,

Ive recently taken interest in this feature but i do have one question. How do i set up the array (i know the array of coordenates)? If im not mistaken (which i must be) we would need a 2 dimensional array for both X and Y to form a rectangle (or a square for that matter, right?

And to clarify, its use would be for MWO as part of an automated script to inform of heat levels on the scale (by running 3 loops for each stage of the bar).

Thanks in advance for your help :)

Exergist

  • Moderator
  • Sr. Member
  • *****
  • Posts: 325
  • Can you dig it?
Re: Get Pixel Color
« Reply #2 on: June 20, 2018, 01:29:58 pm »
Quote
its use would be for MWO
I believe you are referring to MechWarrior Online, correct? If YES then I just want to reiterate that obtaining this kind of data automatically via VA may be considered a terms of service violation. I don't want to take the wind out of your sails, but I also want to make sure you're fully aware of this possibility. The issue probably won't be that you're scanning your monitor (X,Y) coordinates but rather what you'll do with the data you collect. It looks like MWO is competitive, and having this kind of data might be considered a competitive advantage. The only circumstances I would think you wouldn't have an issue is in a single-player non-competitive environment (no rankings, racing, etc). Might even be a good idea to contact the developer and explain what you're trying to do to get a more concrete answer. Or if you don't care, then whatevs. Just my thoughts :)

Onward.

So my example analyzes the pixel directly under the mouse. This pixel corresponds to a specific (X,Y) coordinate within your display(s). I'm guessing this gauge you're looking to monitor has a static size that changes color depending on the heat level. That simplifies the problem since then you just need to figure out an origin pixel coordinate to start your "scan" and then create an algorithm to iterate across the pixels that comprise the shape you want to scan. You'll also need to identify the RGB (or hex) values that correspond to the different colors you may encounter on the gauge.

Also keep in mind that your gauge shape and color characteristics may simplify the problem further. I took a screenshot from a MWO video:



So it looks like you're basically going to need to do a simple 1 pixel wide vertical search and look for where the gauge goes from black to NOT black. Or perhaps you can just monitor the red value of the RGB and once it goes above a certain amount you've "found" the heat level. I also included an arbitrary example along with the calculation (yeh math!) to get a heat level percentage.

So once you've identified (X0,Ymax) and (X0,Ymin), which VA can help you gather, you just have to start your search at (X0,Ymax) and use a loop to subtract 1 from the Y-coordinate, get the pixel data, evaluate the data, and repeat until you find the heat level or you reach Ymin. Then you can just calculate the heat level percentage. You could also start at Ymin and work your way up, though I figure you're probably more likely to be above 50% heat than below it so you'll find the level faster by starting at the top.

Hopefully that helps!
« Last Edit: June 20, 2018, 01:35:51 pm by Exergist »

Tumbler

  • Newbie
  • *
  • Posts: 10
Re: Get Pixel Color
« Reply #3 on: June 20, 2018, 02:16:23 pm »
Exergist,

Damn, you never cease to amaze me. First of all, i didnt even except such a level of indept analysis. But WOW. I actually smiled while reading your post.

So first things first. Yes, it is for mechwarrior online. In all honesty i love voice attack and all your dedication to supporting and hearing the community, so might aswell be as forthcoming as i can. I already contacted by email Piranha games to check if this violates TOS, even if it is for normal non Comunity Warfare (competition gameplay).

Second, JUDAS! I love math!  :o
After posting i actually realized my X axis would be the constant and the only variable to account for would be the Y. Thought tracking the red pixels was (and still is) the original plan. The problem was looking establish an array on C# when in fact i can do just that, reduce the coordinates on the Y axis during the loop... Elegant solution, Exergist!!!!

Now, i will get to working on the Loop settings while i wait for Piranha games response.

The only issue that occured to me was the verbose response. While the pixel is not red, the system only loops to search that red pixel. The problem is, if a verbose response is envoked, it would have to include pauses to keep it from spamming the audio channel, therefore delaying the responsiveness of the loop to any changes. I could circunvent the problem by having an verbal enquiry but that would kind of defeat the automatation of the system...

Exergist

  • Moderator
  • Sr. Member
  • *****
  • Posts: 325
  • Can you dig it?
Re: Get Pixel Color
« Reply #4 on: June 20, 2018, 10:33:26 pm »
Glad to help.

So unless there is a way to grab data from the game engine you'll have to make do with the looping to get updated heat levels. You'll need to judge what works best for you in terms of balancing data updates and keeping the data polling at a reasonable level. You may find that the "level finder" runs very quickly and allows for 1 or 0.5 second pauses while still providing useful flow of data.

Another thought. I recall seeing the level meter flash under certain conditions (overheat?). As a result you may actually want to search for the pixel that transitions from black to another color.

Also you could probably improve your search time by starting at the top and then scanning down for the first iteration. Then once you find the level you could start the next iteration at the pixel where the level was previously found. So if you scan down and find more non-black pixels then you can abruptly change direction and scan upwards.
« Last Edit: June 21, 2018, 10:06:35 am by Exergist »

Tumbler

  • Newbie
  • *
  • Posts: 10
Re: Get Pixel Color
« Reply #5 on: June 21, 2018, 07:40:38 am »
Yeah, gotcha there Exergist. If i use your attached script with the inline, i can check to see what prevalent colours are on the pixels in question. Also, since some times less is more, i changed the function to look in increments of 10% rather than 1%, starting from the top and looking only until the mid way is reached [Y(max)/2], including TTS sentences for 70 and 90% respectively.

Ok i think im on the right path, just fine tunning left. Thanks for getting me there Exergist! Couldnt have done it without you

Exergist

  • Moderator
  • Sr. Member
  • *****
  • Posts: 325
  • Can you dig it?
Re: Get Pixel Color
« Reply #6 on: June 21, 2018, 10:11:07 am »
Glad you're rolling!

The inline function capability is super powerful. And often you can condense lots of lines of VA actions into a single inline function. Good luck!